1. Field of the Invention
This invention relates generally to network switching devices and more particularly to Fibre Channel switching devices and the credit sharing for Fibre Channel links with multiple virtual channels.
2. Description of the Related Art
The Fibre Channel family of standards (developed by the American National Standards Institute (ANSI)) defines a high speed communications interface for the transfer of large amounts of data between a variety of hardware systems such as personal computers, workstations, mainframes, supercomputers, storage devices and servers that have Fibre Channel interfaces. Use of Fibre Channel is proliferating in client/server applications which demand high bandwidth and low latency I/O such as mass storage, medical and scientific imaging, multimedia communication, transaction processing, distributed computing and distributed database processing applications. U.S. Pat. No. 6,160,813 to Banks et al. discloses one of the state of art Fibre Channel switch systems, which is hereby incorporated by reference.
One or more interconnected switches form a network, called a fabric, which other devices, such as mass storage devices, servers or workstations, can be connected to Any devices connecting to a fabric can communicate with any other devices connected to the fabric. A direct connection between two devices is a link. An interface on a device for connecting another device is a port. A non-switch device connecting to a fabric is a node on the network or fabric. A port on a non-switch and non-hub device is an N-port. A port on a switch may be an E-port, for connection to another switch port, an F-port, for connection to an N-port, an FL port for connection to an FC-AL loop or any combination of the above. A link between two switches is an inter-switch link (ISL).
Each port has a transmitter and a receiver. The transmitter sends out information and the receiver receives incoming information. There are buffer memories associated with each port, either the transmitter or the receiver, to temporarily store the information in transit, before the information is confirmed to be transmitted towards its destination by a switch, or to be stored or used by a device at its destination. The buffer memory can be in the actual port or, preferably, may be centralized, as shown in U.S. Pat. No. 6,160,813. The buffer memory is broken down into units. One unit of buffer memory, which can hold one frame, is represented by one buffer-to-buffer credit or one credit. A frame is a unit of information transmitted, which comprises a header portion and a payload portion. The header portion identifies the frame, including a Source Identification (SID) and a Destination Identification (DID). The payload portion contains the data being transmitted. A frame payload may be 2112 data bytes long, which, plus header, CRC, EOF totals 2148 bytes.
In the prior art, a receiver on a port is allocated a fixed amount of buffer space to temporarily store received frames, represented by a fixed number of buffer-to-buffer credits. The receiver controls the allocation of the buffer space. At the initial configuration when two switches connect, the receivers advertise to the transmitters the amount of buffer space represented by the number of credits available. The transmitters initialize their credit counters to the number of credits advertised by the receivers. Both the transmitting port and receiving port keep track of the use of the buffer space using the number of credits and credit counters. Each time a frame is received by the receiver, the frame is stored in a buffer space and the number of credits residing in the receiver is increased by one. The transmitting port keeps track of this by reducing its transmitter credit counter, which indicates how many more frames can be sent, while the receiver increments its receiver credit counter, which indicates how many frames are stored in the buffer space. Once the frame is confirmed to have been retransmitted by a transmitter on the receiving switch, or used by a device, then the buffer space is free to be used to store a new frame. At that time, a credit is returned by a transmitter on the receiving port to a receiver on the transmitting port and the receiver credit counter in the receiving port is decreased by one. When the transmitting port receives the credit, the transmitter credit counter in the transmitting port is increased by one.
Even though frames travel through the fiber optics at the speed of light, it still takes time for frames to move from one device to another. It also takes time for a device to receive a frame; process it or retransmit it; and then return a credit, i.e. a confirmation of receipt, back to the transmitting port. It takes some more time for the credit traveling through the optical fiber to reach the transmitting port. During the turn-around time between when the transmitting port sends out a frame and the transmitting port receives a credit, the transmitting port may have sent out a number of frames at its transmitting speed if the transmitting port has available credits. When the transmitting port has at least a minimum number of credits to allow the transmitting port to continue transmitting until it receives credits back from the receiving port, the effective frame transmission rate is the highest. If the transmitting port does not have that minimum number of credits, then it has to temporarily stop sending frames when all the credits are used and wait until the credits return. Due to this stoppage, the effective frame transmission rate may be substantially lower than the actual transmission rate. That minimum number of credits depends on the turn-around time and the frame transmitting speed. The longer the transmission line, or the faster the transmitting speed, the more frames that may be in transit. At a fixed transmitter speed, the more credits a port can have, the longer the transmission distance can be while the link still maintains the full effective transmitter speed.
In certain switches in the prior art, particularly those using a shared buffer space, there may be a pool of free buffer space which can be used by any ports on the switch needing additional buffer space. A receiving port requests an additional buffer from the pool and transmits an additional credit to the linked transmitting port before the frame is retransmitted from the receiving switch, thus allowing the switch to buffer an additional frame. The credits in this port-level pool are not advertised to the transmitting port. These credits are unknown to the transmitting ports and can only be utilized by the receiving port. When the frame is confirmed out of the receiver, its buffer is released and a credit is returned to the pool, thus allowing other ports on the switch to use the pool buffer space. The port-level pool would be utilized when the frames received from an upstream device are more than the frames retransmitted to a downstream device, for example when the downstream transmission speed is slower than the upstream transmission speed. Another example is when there is a blockage somewhere in the fabric, such as a device holding a loop open, so that frames cannot be delivered.
A switch may have many ports, e.g. 8, 16 or 64 ports etc., to interconnect many devices. Every device connecting through a single switch to many other devices can communicate with each one of them at the full transmission speed of a port-to-port link if there are enough credits on the ports. Two devices may be connected to two different switches and the two switches are directly connected, then the two devices can communicate with each other through the three links in series: device-to-switch, switch-to-switch (ISL) and switch-to-device. Two devices may also communicate with each other through more intermediate ISLs if the two switches they directly connect to do not share a common ISL. When more than one pair of devices communicates through the same ISL, conflicts may arise. For example, one pair of devices may utilize or hog all of the buffers, or other resources, on the ISL, blocking the other pairs of devices from utilizing the ISL.
To address this problem, Brocade Communications Systems Fibre Channel switches logically split the physical ISL into a series of virtual channels (VCs). The data flow can then be segmented among the VCs. The buffer space or credits available to the receiver is thus allocated among the VCs so that the blocking problem discussed above does not occur. Each channel will have at least some buffer space available, and so will have some ongoing data flow, albeit potentially at a reduced rate. VCs are also described in U.S. patent application Ser. No. 60/286,213, entitled “Quality of Service Using Virtual Channel Translation,” by David C. Banks and Alex Wang, filed Apr. 24, 2001, which is hereby incorporated by reference.
In some prior art switches, the buffer space allocation among the VCs has been fixed. The fixed VC credit allocation may result in performance problems, though not as severe as would be present without the use of VCs. If a VC is not in use, then the buffer space allocated to it by the receiver or the credits advertised by the receiver for this VC to the transmitter will not be used. The VCs in use may have their effective transmission rates reduced substantially due to the lack of credits.
Besides allocating all credits to VCs as in some prior art switches, in short distance applications, a receiving port may reserve some credits in a pool, a VC-level pool, available to any of the VCs in the port and evenly allocate the remaining credits to VCs. These credits in the pool are not advertised to the transmitter. When a VC in use is busy, a credit from the pool may be returned to the transmitting port before a buffer for a frame for the VC is released, i.e. ready to be used by next frame. This effectively increases the credits available for a particular VC in use and reduces credits available for a VC not in use. These credits may also be used to smooth the frame/credit flow due to frame traffic congestion, blockage etc. While the VC level pool on a receiver may provide many benefits, it has a big drawback. Due to the reserved credits in the pool, there are less credits advertised overall and less credits advertised for each individual VC to a transmitting port, so the distance concerns are actually exacerbated.
While VCs in the prior art are very beneficial in many situations, they do not help in many long distance situations. As noted above, the VC-level credit pooling actually makes the long distance problem worse. Basically the long distance problem of the prior art is that a receiver must advertise a minimum number of credits, otherwise, the transmitter has to stop sending frame and wait periodically between the time it uses up all the advertised credit until it receives a returned credit. The more credits advertised to a transmitter, the less the long distance problem. The more credits actually available to a transmitter, the less the long distance problem. If the credits advertised or available to a transmitter are less than the needed minimum, the effective transmission rate drops substantially, as described above. The use of VCs, and to a greater extent VCs with a credit pool, actually reduces the total credits available to a transmitting port and the credits available to VCs. This long distance problem is illustrated in more detail below.
This problem could be resolved by providing additional buffer space for each receiver, but in many cases the additional buffer space would not be utilized, thus wasting scarce resources on the ASICs which comprise the receivers. Therefore, it is desirable to reduce the long distance problem but at the same time conserve ASIC resources. It is also desirable not to cause new problems.